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This  report  reviews  the  psychological  literature  on  the  effects  of  answering 
questionnaires  on  individual  behavior.  First  it  reviews  the  data  on  general 
ized  criticisms  of  questionnaires.  Next  it  reviews  experimental  studies 
dealing  with  the  effects  of  exposure  to  questionnaires  on  subsequent  atti¬ 
tudes  of  the  individual.  Then,  studies  of  the  use  of  questionnaires  by 
organizations  and  their  behavioral  effects  are  reviewed.  Finally,  the  im¬ 
plications  of  these  results  for  research  at  USMA  are  discussed,  and  several 
recommendations  are  made. 
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INTRODUCTION 


Every  year  the  USMA  Office  of  Institutional  Research  designs  and/or  administers  a  number 
of  research  questionnaires.  These  questionnaires  generally  elicit  a  wide  variety  of 
reactions  both  from  cadets  and  from  members  of  the  staff  and  faculty.  In  the  process  of 
planning  the  questionnaires,  Academy  officials  frequently  challenge  the  potential  use¬ 
fulness  of  certain  items,  or  express  the  concern  that  certain  questions  despite  their 
potential  U8cfulness--may  have  an  undesirable  effect  on  the  cadets.  This  concern  that 
research  questionnaires  may  have  potentially  harmful  effects  is  an  entirely  legitimate 
concern  of  the  administrative  leadership  of  any  organization  which  uses  such  question¬ 
naires.  The  professional  psychologists  who  design  such  questionnaires  also  are  legiti¬ 
mately  concerned,  both  in  terms  of  professional  ethics  and  in  terms  of  the  accuracy  of 
the  data  they  obtain. 

Principle  7  of  the  Ethical  Principles  in  the  Conduct  of  Research  with  Human  Participants 
puts  the  matter  clearly:  "The  ethical  investigator  protects  participants  from  physical 

and  mental  discomfort,  harm,  and  danger.  If  the  risk  of  such  consequences  exists,  the 
investigator  is  required  to  inform  the  participant  of  that  fact,  secure  consent  before 
proceeding,  and  take  all  possible  measures  to  minimize  distress.  A  research  procedure 
may  not  be  used  if  it  is  likely  to  cause  serious  and  lasting  damage  to  participants.” 
Thus,  administrators  and  research  psychologists  agree  on  the  importance  of  knowing  more 
about  the  actual  effects  of  research  questionnaires. 

This  retort  deals  with  the  factual  issue:  What  is  known  about  the  actual  effects  of 
research  questionnaires  on  persons  who  take  them?  Psychological  abstracts  were  con¬ 
sulted  for  psychological  theories  or  empirical  research  which  might  bear  on  the  problem. 
Questions  such  as  the  following  were  asked:  Do  questionnaires  cause  people  to  think 
more  seriously  about  the  topics  concerned?  Do  organi rations  which  use  questionnaires 
produce  lower  quality  outputs  than  organizations  which  do  not?  In  retrospect,  do  per¬ 
sons  who  have  engaged  in  socially  undesirable  behavior  recall  greater  exposure  to  re¬ 
search  questionnaires?  Do  organizations  which  use  questionnaires  continually  know  of 
cases  where  the  survey  provoked  an  increase  or  decrease  in  the  behavior  under  study? 

The  last  of  these  questions  is  perhaps  the  most  important.  If  an  affirmative  answer  is 
given,  then  professional  ethics  would  compel  us  to  place  severe  restrictions  on  the  use 
of  such  surveys. 

Before  turning  to  the  body  of  the  report,  which  is  organized  around  the  questions  listed 
above,  a  few  definitions  and  limitations  of  this  report  should  be  noted.  By  "research 
questionnaires”  we  mean  questionnaires  written  by  qualified  psychologists,  sociologists, 
or  political  scientists.  "Questionnaires"  are  sometimes  used  by  Congressmen,  for  exam¬ 
ple,  to  impress  voters,2  or  by  advertisers  to  increase  product  awareness.  Questionnaires 
are  sometimes  written  and  reported  to  bolster  a  particular  point  of  view.  The  present 
report  deals  only  with  the  effects  of  professionally  constructed  questionnaires. 

Propagandists  can  write  questions  which  almost  compel  a  person  to  give  the  desired 
answer.  They  may  then  use  the  results  of  the  survey  to  "prove”  that  the  public  supports 
their  point  of  view.  Such  questionnaires  are  not  considered  "research  questionnaires" 
in  the  body  of  this  report.  This  report  will  focus  on  documented  psychological  theories 
and  documented  effects  of  professionally  written  research  questionnaires. 


^American  Psychological  Association,  Washington,  DC,  1973. 

2A  number  of  entertaining  examples  are  given  by  Leo  Bogart  in  Silent  Politics:  Polls 
and  the  Awareness  of  Public  Opinion,  New  York,  Wiley,  1972. 

\  good  check  list  of  things  to  avoid  in  questionnaire  construction  is  provided  by 
Stanley  L.  Payne  in  The  Art  of  Asking  Questions,  New  Jersey,  Princeton  University 
Press,  1951. 


CRITICISMS  OF  QUESTIONNAIRES:  ANXIETY,  STRESS.  AND  INTEREST 


Th«r«  l*  an  impressive  amount  of  evidence  that  ordinary  classroom  tests  cause  students 
at  all  a<)«  levels  to  experience  fear,  excessive  perspiration,  and  perhaps  even  some 
impairment  of  performance.  Of  course,  classroom  tests  are  used  to  evaluate  individuals 
and,  for  that  reason,  are  potentially  more  threatening  to  persons  than  are  research 
questionnaires.  The  individual  who  takes  part  in  a  research  survey  is  told  "this  is  not 
a  teat.  The  results  will  not  affect  your  official  record  at  this  institution.  Vour  an¬ 
swers  are  confidential."  The  purpose  of  such  instructions  is  to  prevent  mental  stress 
and  to  encourage  honest  responses.  To  the  extent  that  researchers  keep  confidences  and 
the  respondents  believe  them,  the  research  questionnaire  will  be  less  stressful  than  the 


Even  the  most  carefully  designed  resoarch  questionnaire,  however,  does  occasionally 
stimulate  complaints.  The  frequency  of  complaints  about  a  questionnaire  is  an  index  of 

Th”  next  few  P*r*IJr*Ph»  outline  the  results  of  studies  which  deal 
with  subject  complaints,  or  reactions  to,  research  questionnaires. 


The  Program  Evaluation  Office  of  the  US  Office  of  Education  did  a  study  of  50  consecu¬ 
tive  research  projacts  submitted  to  them.  Eight  criteria  were  used  to  evaluate  each 
questionnaire  item  for  clearance)  for  example,  "Does  the  item  call  for  self-incrimi¬ 
nating  or  self-debasing  admission  or  confession,"  or  "Does  the  item  seem  to  countenance 
(or  give  unduly  neutral  recognition  to)  behavior  or  views  which  are  generally  considered 
highly  reprehensibie,  immoral,  contrary  to  public  policy,  etc."5  In  the  50  projects 
studied,  there  were  109  questionnaires  with  cor  5300  separate  items.  Of  these,  only 
10  items  were  judged  objectionable  by  the  USQS  Clearance  Comi.il  tee.  These  figures  are 
not  the  result  of  cursory  or  indulgent  examination  of  the  instruments"  (p.  358) . 

The  US  Commission  of  Obscenity  and  Pornography  conducted  a  national  survey  of  public 
attitudes  toward  erotic  materials.  Because  such  questions  are  generally  considered 
sensitive,  one  might  expect  a  high  frequency  of  complaints  about  the  questionnaire. 

The  interviewers  for  this  study,  mostly  female,  were  trained  for  three  days  before 
going  out  into  the  field.  Respondents  were  contacted  by  letter  prior  to  the  interview 
Surprisingly,  negative  comments  about  the  interview  were  infrequent.6  After  the  inter- 
View,  ammh  subjectsprivately  completed  a  tWee-item  questionnaire  designed  to  measure 
the  candor  with  which  they  had  answered  the  interviewer.  Candor  varied  somewhat  with 
town  sixe,  church  attendance,  education,  and  reported  exposure  to  erotica. 


Both  the  US0E  study  and  the  Pornography  Commission  study  are  important  because  they 
eait  with  highly  sensitive  issues.  They  demonstrate  that  professionally  written  re¬ 
search  questionnaires  generate  very  little  antagonism  in  those  who  are  questioned. 

Although  there  has  been  some  criticism  of  "testa"  in  the  popular  press,  much  of  the 
criticism  has  been  directed  against  aptitude  tests  or  intelligence  tests)  there  are 
fewer  complaints  about  professionally  written  research  questionniares. 


The^concept  of  test  anxiety  is  reviewed  by  Secord  and  Backman  in  A  Social  Psychology 
~  Th!,^TPt  °f  te9t  rasl8tanc“  recently  measured  by  C.  Cuininghii, 
AbstMcts  ^966  27  1651  UniVersity  Hl<}h  Schocl  students."  Dissertation 


H.  S.  Conrad,  "clearance  of  Questionnaires  with  Respect  to  'Invasion  of  Privacy  • 
Public  Sensitivities,  etc.,"  American  Psychologist.  1967,  22,  356-360. 

6 

L.  Losciuto,  A.  Spertor,  E.  Michaels,  and  C.  Jenne,  Methodological  Report,  in 
Technical  Report  of  the  Conroiaaion  on  Obscenity  and  Pornography,”  Volume  VI, 
National  Survey  of  Public  Attitudes  Toward  and  Experience  with  Erotic  Materials 
US  Printing  Office,  1970.  ‘  ‘  - - - - ■* 


Humanist  critics  like  Joseph  Wood  Crutch7  seem  to  believe  that  certain  human  qualities, 
like  happiness,  cannot  be  measured.  Paul  Goodman,8  on  the  other  hand,  criticizes  one 
sociological  questionnaire  for  not  measuring  important  qualities:  “He  never  asks  a 
question  which  takes  a  youngster  seriously  as  a  human  being...  There  is  no  way  offered 
to  express  one's  real  sentiments  on  loyal  friendship,  love,  knowledge,  pride  in  a  useful 
job,  moral  courage,  radical  dissent,  fear  and  trembling. . .  What  strikes  me — and  what 
must  vaguely  strike  the  teenager-ris  that  the  avoidance  of  these  questions  is  a  signal" 
(p.  279-280).  Research  psychologists,  of  course,  do  not  try  to  measure  every  important 
human  attribute  in  every  survey.  Furthermore,  the  need  for  brevity  and  standardization 
in  research  questionnaires  often  does  conflict  with  poetic  nuances  of  expression  which 
are  used  by  the  great  philosophers. 

The  most  serious  public  criticism  of  psychological  tests  dealt  with  the  use  of  person¬ 
ality  tests  in  selecting  applicants  for  government  jobs.  One  such  test,  the  Minnesota 
Multiphasic  Personality  Inventory  (MMPI)  originally  was  designed  to  help  psychiatrists 
diagnose  the  mentally  ill.  The  MMPI  Paranoia  Scale,  for  example,  consists  of  40  state¬ 
ments  like  "no  one  understands  me."  If  an  individual  endorses  an  above  average  number 
of  such  statements,  he  might  be  diagnosed  as  paranoid.  In  1965,  the  MMPI  and  similar 
tests  were  used  by  the  State  Department  for  screening  persons  going  to  overseas  posts 
where  emotional  stability  might  be  of  prime  importance .  Similar  questions  were  used  by 
Department  of  Defense  to  assure  that  "unreliable  persons  did  not  get  into  posts  where 
their  instability  could  be  dangerous."8  Critics  objected  to  "invasion  of  privacy." 

Many  psychologists  also  objected  to  the  use  of  tests  for  purposes  other  than  originally 
intended.  Technically,  a  test  which  is  valid  for  diagnostic  purposes  in  a  hospital 
setting  is  not  necessarily  valid  for  selection  purposes.  No  new  legislation  resulted 
from  the  1965  Congressional  hearings  on  this  issue,  and  psychological  tests  are  still 
being  used  in  the  Federal  Government.  The  issue  is  still  somewhat  controversial.^8 

The  next  few  paragraphs  report  more  systematic  investigations  of  how  the  subject  reacts 
to  psychological  research  questionnaires.  Instead  of  relying  on  spontaneous  comments, 
all  subjects  were  asked  specifically  for  their  reactions  and  evaluation  of  a  research 
questionnaire. 

In  one  experiment,  individuals  were  asked  24  questions  which  were  classified  as  psy¬ 
chologically  "intimate"  or  as  "non- intimate"  questions.  Prior  to  responding  to  these 
questions,  the  research  subjects  were  asked  to  play  the  role  of  an  interviewee  in  one 
of  three  different  settings.  The  three  different  interview  "settings"  were  created  by 
having  the  "interviewee”  listen  to  a  tape  recorded  interview  of  (a)  a  business  inter¬ 
view,  (b)  a  psychiatric  interview,  or  (c)  a  sex  research  interview.  After  responding 
to  each  question,  the  "interviewee”  rated  the  interviewer-in- training.  As  you  might 
expect,  the  interviewees  rated  the  interviewer  less  favorably  after  an  intimate  ques¬ 
tion  than  after  a  non-intimate  question.  The  results  also  showed  that  the  interview 
setting  influenced  response  to  the  interviewer.  In  a  business  context,  intimate  ques¬ 
tions  produced  the  most  disliking,  whereas  the  sex  interview  context  produced  the  least 


Crutch,  J.  W. ,  "Calipers  on  the  Human  Mind,"  Saturday  Review,  19  Jun  65. 

'Goodman,  P. ,  The  Community  of  Scholars,  New  York:  Vintage  Books,  1962. 

Amrine,  M. ,  The  1965  Congressional  inquiry  into  testing:  A  commentary,  American 
Psychologist,  1965,  20,  p.  867. 

°The  issue  of  invasion  of  privacy  is  complex,  especially  since  the  US  Constitution 
does  not  refer  specifically  to  the  concept  of  privacy.  In  "What  Price  Privacy," 

C.  C.  Bennett  agrees  that  the  right  to  maintain  personal  secrets  conflicts  with  an 
organization's  need  to  build  trust  and  to  protect  itself.  See  C.  C.  Bennett,  "What 
Price  Privacy,"  American  Psychologist,  1967,  22,  371-376. 
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disliklng--even  though  the  same  questions  were  asked  in  either  setting.11  This  study 
suggests  that  attitudes  toward  research  questionnaires  may  depend  upon  the  purposes  of 
the  research  as  the  subject  understands  them. 

Ur .  Donald  W.  Fiske  completed  two  studies  of  the  subject's  reaction  to  psychological 
teats.  The  first  study  was  a  national  opinion  survey  of  the  American  public.  Respon¬ 
dents  were  asked  to  take  two  short  "personality  tests"  selected  from  a  set  of  six 
abbreviated  scales;  an  interest  inventory,  an  attitude  scale,  and  a  test  of  psycho¬ 
pathology  were  among  those  included.  Before  taking  the  two  psychological  tests,  half 
of  the  respondents  were  told  to  imagine  they  were  applying  for  a  job,  and  the  other 
half  were  told  to  imagine  they  were  taking  these  tests  for  "research”  purposes  only. 
There  were  few  differences  in  response  to  these  two  different  settings.  Although  there 
was  a  wide  variety  of  reactions  to  these  tests,  most  persons  said,  "I  don't  mind,"  or 
“I  was  interested,"  or  “I  was  curious"  in  response  to  the  tests.  Risks  concluded  that 
the  general  public  is  not  too  concerned  one  way  or  the  other  with  psychological  test¬ 
ing  (p.  294)  .l  2 

In  a  later  study,  Riske  explored  several  additional  factors.  College  students  took  a 
personality  teat  under  anonymous  conditions  and  answered  a  series  of  open-ended  ques¬ 
tions  concerning  their  reactions  to  the  test.  Some  subjects  took  a  rather  disorganised 
questionnaire  with  items  selected  at  random  from  several  standard  personality  tests. 
Other  subjects  took  a  longer  test  with  four  sub-scales  selected  from  standard  scales. 

A  third  group  took  a  complete  standard  personality  test.  The  comments  of  all  three 
groups  were  classified  into  five  categories:  (a)  desire  for  information  about  the  test 
or  tester,  <b)  desire  for  information  about  self,  (c)  criticisms  of  testing  of  item 
repetitiveness  or  ambiguity,  (d)  criticisms  of  self,  (e)  apprehensions  and  fears.  Sub¬ 
stantial  proportions  of  subjects  wanted  more  information  about  the  test  and  were 
critical  of  testing  in  the  first  two  groups,  in  the  third  group,  subjects  manifested 
more  desire  to  learn  about  themselves  and  found  the  tests  more  interesting.  Fiske 
noted  a  wide  variety  in  individual  reactions  to  tests.  In  conclusion,  he  recommends 
providing  the  subject  with  more  information  about  the  purpose  of  the  test  and  more 
feedback  about  the  results  in  personality  research. 

For  many  years,  the  IBM  Corporation  surveyed  the  attitudes  of  its  employees.  Since 
opinion  surveys  were  used  more  widely  in  the  I960' s  than  they  were  10  years  before, 
management  Instituted  a  program  of  evaluation.14  Both  manufacturing  employees  and 
managers  were  asked:  (a)  their  degree  of  satisfaction  with  feedback  from  the  survey 
and  (b)  the  degree  to  which  they  thought  the  results  of  the  survey  were  being  utilised. 
Both  written  reports  and  open  meetings  were  used  to  report  back  to  survey  respondents 
on  the  results  of  research.  Feedback  meetings  were  preferred  to  written  reports  as  a 
means  of  learning  about  the  results  of  the  research.  People  were  more  satisfied  with 
the  results  of  the  research  if  they  received  information  about  their  own  unit  in  the 
organisation.  For  the  line  employees,  satisfaction  with  feedback  was  independent  of  the 
perceived  utilisation  of  the  survey  results;  for  managers,  however,  satisfaction  and 
utilisation  were  closely  lined.  Managers  want  opinion  survey  results  to  be  used.15 


Vaplan,  K.  J.,  Firestone,  1.  J.,  Moore,  M. ,  and  Degnore,  R.,  Attitude  toward  inter¬ 
viewer  as  a  function  of  question  intimacy  across  three  interview  settings.  Pro¬ 
ceedings  of  the  79th  Annual  Convention,  American  Psychological  Association, 

Washington  DC,  1971,  pp.  375-376. 

12 

Fiske,  D.  W.,  The  Subject  Reacts  to  Tests,  American  Psychologist,  1967,  22,  287-296. 

*Viske,  D.  W.,  and  Kuncel,  R.  B.,  How  Does  it  Feel  to  Take  a  Personality  Test?  Pro¬ 
ceedings  80th  Annual  Convention,  American  Psychological  Association,  1972,  7_,  pp.  25- 
26. 

^Kraut,  A.  E.  ,  “Opinion  Surveys:  Turning  Results  into  Action,"  Personnel ,  1966,  43, 
58-65. 

lSKlein,  S.  M.,  Kraut,  A.  E.,  and  Wolfsan,  A.,  "Bnployee  Reaction  to  Attitude  Survey 
Feedback;  A  study  of  the  Impact  of  Structure  and  Process.”  Administrative  Science 
Quarterly,  1971,  16,  497-514. 
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In  summary,  research  questionnaires  do  generate  some  small  criticisms.  Few  subjects  in 
a  national  survey  of  attitudes  to  pornography  criticized  the  interview.  Most  items 
scrutinized  by  USOE  were  inoffensive.  The  use  of  personality  tests  for  job  selection 
has  been  criticized  in  Congress,  but  no  serious  offenses  were  uncovered.  Experimental 
studies  show  people  do  react  more  negatively  to  “intimate"  questions  which  are  posed  rn 
the  context  of  a  business  interview.  The  general  public  is  not  too  concerned  about  per¬ 
sonality  testing,  but  criticism  can  be  increased  by  presenting  badly  organized  questions 
to  the  research  subjects  without  fully  explaining  the  purpose  of  the  research.  In  a 
arge  company,  people  are  more  satisfied  with  questionnaire  research  if  they  are  told 
about  it  in  a  public  meeting,  are  given  data  privately  about  their  own  organizational 
unit,  and  if  they  see  the  results  of  the  research  being  used. 


EFFECTS  OF  QUESTIONNAIRES  ON  ATTITUDE  CHANGE 

A  biased  questionnaire  can  induce  some  opinion  change.  Two  researchers16  in  Texas  illus¬ 
trated  this  fact  by  constructing  three  experimental  questionnaires  measuring  attitudes 

TABLE  1 

ILLUSTRATIVE  ITEMS  FROM  EACH  OF  THE  EXPERIMENTAL  QUESTIONNAIRES 


Form 

%S's  Choosing 
Each  Response 

Lenient 

1. 

Is  a  policy  of  imprisonment  for  all  crimes  a  good  idea 
since  it  would  involve  locking  up  petty  offenders  who 
would  never  try  another  crime,  as  well  as  persons  who 
had  one  weak  moment,  or  were  victims  of  circumstance? 

a.  Yes,  any  person  who  commits  a  crime,  no  matter  how 
petty,  should  be  put  in  prison. 

10 

b.  No,  a  policy  of  imprisonment  for  all  crimes  is  im- 
Practical  as  well  as  unreasonable. 

90 

Harsh 

1. 

Is  a  policy  of  imprisonment  only  for  "serious"  offenses 
or  habitual  criminality  a  good  idea  when  it  would  probably 
encourage  petty  offenders  and  criminals  who  had  never  been 
caught  to  continue  in  their  crimes? 

a.  Yes,  a  policy  of  punishment  for  very  serious  offenders 
only,  is  a  good  idea. 

0 

Neutral 

b.  No,  it  doesn't  make  good  sense  to  adopt  a  policy  which 
ways,  in  effect,  "small  jobs  are  okay  as  long  as  you 
are  not  caught  pulling  them  off  more  than  a  couple  of 
times . " 

100 

1. 

Is  a  policy  of  imprisonment  only  for  serious  offenses  or 
habitual  criminality  a  good  idea? 

a.  Yes,  a  policy  of  imprisonment  for  serious  offenders, 
or  habitual  criminals  only,  is  a  good  idea. 

54 

b.  No,  such  a  policy  is  not  a  good  idea. 

46 
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to  the  treatment  of  criminals.  One  questionnaire  was  slanted  so  as  to  elicit  "lenient" 
attitudes;  another  was  slanted  to  elicit  “harsh"  attitudes;  and  the  third  was  unbiased 
or  “neutral."  Table  1  gives  an  example  of  the  type  of  item  used,  and  shows  that  re¬ 
search  subjects  did  give  the  type  of  response  which  each  type  of  biased  item  demanded. 

The  subjects  were  divided  into  three  different  groups,  and  each  group  took  a  different 
questionnaire.  Then  they  all  took  the  same  attitude  scale,  which  was  a  professionally 
constructed  unbiased  measure  of  attitude  to  the  treatment  of  animals.  Males  who  took 
the  "harsh"  pre-test  scored  75  on  the  common  post-test)  those  who  took  the  "lenient" 
pre-test  scored  93  on  the  common  post-test)  and  those  who  took  the  "neutral"  pre-test 
scored  78.  The  difference  between  these  group  means  is  statistically  significant  and 
a  counts  for  13%  of  the  variance  of  subjects’  scores.  The  "lenient"  pre-test  resulted 
in  larger  differences  than  the  "harsh"  pre-test. 

The  research  did  not  establish  how  long  the  bias  would  last,  since  the  post-test  was 
taken  immediately  after  the  biased  questionnaire.  Social  psychological  studies  of 
opinion-change  generally  show  that  the  impact  of  a  persuasive  message  decays  with  time. 
If  the  "biased"  pre-test  questionnaire  functions  like  propaganda,  its  impact  would  be 
greatest  immediately  after  exposure  to  it  and  would  diminish  with  time.  Thus,  a  "13% 
change  in  attitude  responses"  probably  over-estimates  the  long  term  effects  of  a  biased 
pre-test  on  subsequent  attitudes. 

Further  parametric  studies  of  this  phenomenon  would  help  us  use  the  results  wisely.  In 
ordinary  practice,  it  woj^d  be  difficult  to  construct  pre-tests  as  biased  as  the  ones 
constructed  by  Dillehay:  His  harsh  and  lenient  pre-tests  represent  extremes  on  a  con¬ 

tinuum  of  questionnaire  biasing.  Professional  researchers  try  to  be  "unbiased"  in 
constructing  questionnaires:  if  their  questionnaires  are  biased,  the  bias  is  uncon¬ 
scious,  and  less  extreme.  Social  psychological  studies  of  opinion-change  generally  show 
that  extreme  appeals  have  a  greater  persuasive  impact  than  moderate  appeals,  assuming 
the  message  is  credible.  Thus,  if  biased  questionnaires  function  like  persuasive  mes¬ 
sages,  the  more  extreme  the  bias,  the  greater  the  persuasive  impact. 

Do  unbiased  questionnaires  cause  attitude  change?  A  number  of  recent  studies  have  in¬ 
vestigated  the  effects  of  giving  a  pre-test  on  response  to  persuasive  messages.  Table  2 
gives  an  illustration  of  the  basic  experimental  design  used  in  a  number  of  such  studies. 


THE  FOUR-GROUP  DESIGN 

TABLE  2 

FOR  EXPERIMENTS  ON  ATTITUDE 

CHANGE 

TIME 

GROUP 

First  Measurement 
Period 

Intervening  Period 

Second  Measurement 
Period  (Post-test) 

A 

Take  an  attitude 
scale  (pre-test) . 

* 

Hear  a  persuasive 
message . 

Take  the  same  atti- 
titude  scale  again. 

B 

Wait)  do  nothing. 

Hear  a  persuasive 
message. 

Take  the  attitude 
scale. 

C 

Take  an  attitude 
scale  (pre-test) . 

Wait)  no  message. 

Take  the  same  atti¬ 
tude  scale  again. 

D 

Wait. 

Wait)  no  message. 

Take  the  attitude 
scale. 
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Individuals  are  assigned  to  each  experimental  group  at  random,  and,  relying  on  the  law 
of  large  numbers,  should  have  equivalent  attitude  means  before  exposure  to  the  two  ex¬ 
perimental  treatment  variables.  Each  of  the  four  groups  receives  one  of  four  possible 
sequences  of  pre-test  (or  no  pre-test)  followed  by  a  persuasive  message  (or  no  message) . 

Since  all  four  groups  start  with  the  same  attitude,  analysis  of  post-test  scores  can  be 
used  to  infer  change  in  attitude. 

In  the  first  study  using  this  design,  college  students  were  pre-tested  (Groups  A&C, 

Table  2)  on  their  attitudes  toward  vivisection.  Twelve  days  later,  some  of  them  heard  a 
lecture  which  argued  strongly  in  favor  of  vivisection.  The  pro-vivisection  lecture  was 
effective  in  changing  attitudes  toward  vivisection,  as  the  results  showed.  Those  who 
were  pre-tested  showed  no  effect  of  pre-testing,  and  the  pre-testing  did  not  affect  the 
reception  given  to  the  lecture.18  A  second  study  with  a  mental  health  film  as  the  per¬ 
suasive  medium  showed  the  same  results:  the  persuasion  worked  to  reduce  ethnic  preju¬ 
dice,  but  pre-testing  had  no  direct  or  indirect  effects  on  post-test  scores.19  A  third 
study,  which  used  arguments  both  for  and  against  vivisection,  produced  similar  re¬ 
sults.20 

One  study  used  a  variation  of  the  experimental  design  given  in  Table  2,  All  of  the 
subjects  read  a  persuasive  message  on  advertising,  some  after  being  pre-tested  for 
attitudes  to  advertising  two  weeks  before.  Half  the  subjects,  after  reading  the  pro¬ 
advertising  message,  but  before  taking  the  post-test,  answered  a  questionnaire  which 
was  designed  to  make  them  suspicious  of  the  purpose  of  the  experiment  and  thus  to  dis¬ 
count  the  "propaganda."  The  results  show  that  the  suspicion  arousing  procedure  was 
ineffective  in  producing  attitude  change.  Furthermore,  there  was  no  effect  of  pre¬ 
testing  on  attitudes  to  advertising  on  the  post-test.21 

Several  studies  using  the  four-groups  design,  or  an  elaboration,  found  that  pre-testing 
has  a  dampening  effect  on  subsequent  opinion  change.  With  vivisection  as  the  topic  of 
persuasion,  using  a  two-sided  lecture,  subjects  who  were  pre-tested  changed  their  opin¬ 
ions  less  than  those  who  were  not  pre-tested.22  In  a  series  of  studies  on  attitudes  to 

0-3 

racial  integration,  "pre-tested  groups  change  less  than  unpre-tested  groups"  (p.  660) . 

Attitudes  toward  making  decisions  often  shift  toward  a  higher  preference  to  risk  after 
group  discussion  of  the  decision  problem.  In  a  recent  study  of  this  phenomenon,  it  was 
discovered  that  taking  a  pre-test  significantly  inhibits  post-discussion  opinion 
change.21* 
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Summarizing  the  results  of  tent experimental  studies.  Nosanchuck  developed  a  theoretical 
model  to  explain  the  results.2^  When  an  individual  takes  an  attitude  scale  for  the  first 
time,  two  things  take  place:  first,  he  is  exposed  to  new  information  which  he  may  not 
have  considered  before;  second,  he  makes  a  response  to  the  attitude  item  and  coirmits  him- 
ae^  *  position.  The  two  events  have  opposing  effects  on  subsequent  attitude  change; 
exposure  to  new  information  tends  to  promote  attitude  change,  whereas  making  a  commitment 
to  a  position  tends  to  inhibit  attitude  change.  In  most  of  the  studies  he  reviewed,  the 
"commitment"  effect  was  the  stronger  of  the  two. 

In  discussing  the  effects  of  pre-testing  on  attitude  change,  one  should  distinguish 
between  attitudes  and  factual  knowledge.  “Attitudes"  imply  the  individual  has  adopted 
them  freely  for  personal  reasons  in  the  absence  of  a  clearly  demonstrable  official  an¬ 
swer.  If  a  teacher  gives  a  “pre-test"  of  factual  knowledge  to  his  class  before  they 
read  a  chapter  in  the  textbook,  they  will  naturally  do  better  on  a  post- test  than  the 
nonpre-tested  students.  In  the  context  of  classroom  learning,  a  pre-test  tells  the  stu¬ 
dents  what  to  study  for  on  the  next  test.  This  obvious  point  has  been  demonstrated  by 
two  experiments:  one  dealing  with  training  of  psychiatric  nurses,26  the  other  dealing 
with  factual  recall  of  a  388  word  paragraph  on  mental  health.27 

Of  course,  it  is  sometimes  difficult  to  demonstrate  whether  a  statement  is  a  matter  of 
factual  knowledge  or  is  merely  a  matter  of  opinion.  In  really  ambiguous  cases,  indi¬ 
vidual  motives  will  influence  the  judgment  that  “x  is  a  fact"  as  opposed  to  "x  is  an 
opinion."  If  a  man  is  highly  motivated  to  please  the  test-giver,  he  may  judge  the  items 
on  the  pre-test  as  factual  statements;  if  he  is  poorly  motivated,  he  may  judge  the  items 
as  opinion  statements.  In  the  first  case,  pre-testing  should  facilitate  attitude 
change;  in  the  second,  pre-testing  should  Inhibit  attitude  change.  Such  a  prediction 
was  supported  in  a  study  by  Rosnow  and  Suls.28  To  create  motivation  to  please  the  ex¬ 
perimenters,  they  paid  some  of  their  subjects  who  volunteered  for  the  experiment.  Other 
subjects  did  not  volunteer  but  were  required  to  participate  in  the  experiment.  A  four- 
group  design  was  used  with  the  volunteers  and  the  non-volunteers.  Attitude  toward 
nuclear  research  and  its  danger  was  the  target  of  the  persuasive  messages  and  the  post¬ 
test.  As  predicted,  pre-tests  immunized  non-volunteers  against  attitude  change;  for  the 
volunteers,  pre-testing  facilitated  attitude  change. 

One  final  note  of  caution  should  be  made  with  respect  to  research  on  the  pre-test  treat¬ 
ment  interaction.  Some  of  the  studies  cited  in  the  report  deal  with  rather  trivial 
issues— such  as  "attitudes  toward  vivisection."  While  there  is  no  empirical  research  on 
the  effect  of  topic  importance,  social  psychologists  believe  that  it  is  easier  to  change 
unimportant  attitudes  than  important  ones.  Thus,  any  effects  of  pre-tests  are  probably 
confined  to  beliefs  about  relatively  unimportant  matters.  Firmly  established,  well  sup¬ 
ported  rational  beliefs  probably  are  not  affected  by  pre-tests. 

In  summary ,  biased  questionnaires  can  cause  attitude  change  while  unbiased  question¬ 
naires  inhibit  attitude  change,  provided  that  the  questionnaires  are  perceived  as  genuine 
research  questionnaires  rather  than  concealed  measures  of  classroom  learning. 
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ORGANIZATIONAL  USE,  CASE  STUDIES,  AND  BEHAVIORAL  EFFECTS 
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In  a  third  study,  the  effect  of  counseling  interviews  on  subsequent  behavior  was  studied. 
Men  undergoing  basic  combat  training  at  an  Army  training  center  took  a  psychological 
test  which  predicts  discipline  problems — "The  AWOL  Syndrome."  About  10%  of  the  men  were 
identified  as  high-risk  cases  using  the  test.  Of  this  group,  half  the  men  were  inter¬ 
viewed  by  their  Commanding  Officer  to  find  out  what  might  be  done  to  prevent  AWOL  or 
other  discipline  problems.  For  the  other  half  of  the  men,  their  tests  were  not  scored 
in  the  field,  and  they  were  not  interviewed  by  their  CO.  Of  those  interviewed,  12% 
were  later  punished  under  the  Uniform  Code  of  Military  Justice.  Of  those  not  inter¬ 
viewed,  only  7%  were  later  punished.  Clearly,  the  "counseling  interview"  used  by  CO's 
had  a  detrimental  effect  on  the  behavior  of  the  men.  The  researchers  term  this  a  case 
of  self-fulfilling  prophecy.35 

In  comparing  the  studies,  the  interviews  in  the  first  two  were  for  research  purposes, 
whereas  the  interview  in  the  third  was  to  prevent  misbehavior.  It  seems  likely  that  in 
the  latter  study,  the  interviewer  wa3  not  merely  recording  attitudes,  but  was  actively 
trying  to  change  them.  In  general,  the  results  of  this  section  show  no  harmful  effects 
of  research  oriented  interviews.  When  questionnaires  are  used  by  organizations  for 
planning  or  for  selection  of  employees,  the  results  are  beneficial  to  the  organization. 


IMPLICATIONS  AND  RECOMMENDATIONS 

We  have  now  completed  our  review  of  the  effects  of  research  questionnaires  on  the  per¬ 
sons  who  take  them.  All  of  the  research  studies  cited  in  this  review  were  done  outside 
the  Military  Academy,  and  most  were  focused  on  non-military  situations.  What  are  the 
implications  of  these  studies  upon  institutional  research  at  the  Military  Academy?  This 
section  will  attempt  to  generalize  from  the  above  cited  extramural  studies  and  refer  to 
local  experience  when  it  is  relevant. 

Three  studies  cited  above  noted  a  low  frequency  of  complaints  about  professional  surveys 
or  a  low  degree  of  public  concern.  Outf experience  at  West  Point  is  congrue*nt  with  these 
extramural  studies.  Dr.  Richard  Butler  found  fairly  good  agreement  among  research  psy¬ 
chologists  and  an  Army  officer  on  the  degree  of  "sensitivity"  of  over  35  questionnaire 
items.  Most  of  these  wore  judged  not  sensitive  or  controversial.  But  even  the  sensi¬ 
tive  items  did  not  cause  anonymous  respondents  and  nan-anonyfcous  respondents  to  differ 
in  their  responses . 3r'  Furthermore,  although  the  average  cadet  takes  more  than  12  sur¬ 
veys  in^is  four  years  at  West  Point,  the  questionnaires  are  seen  as  a  useful  medium  of 
change.  Finally,  a  content  analysis  of  comments  by  First  Classmen  indicates  that 
"Research  Questionnaires"  are  of  little  concern.  Of  22  topics  which  elicited  five  or 
more  comments,  "Research  Surveys"  ranked  15.5  in  frequency  of  mention,  far  below  topics 
like  "Cadet-Tactical  Officer  Relations,"  "The  Discipline  System,"  and  "The  Honor  System." 

While  research  questionnaires  at  West  Point  have  been  remarkably  non-controversial  (at 
least  to  the  subjects) ,  it  may  be  possible  to  improve  them  still  further.  The  following 
recommendations  are  based  on  principles  derived  from  the  preceding  review  of  the  litera¬ 
ture  i  the  principle  of  giving  cadets  a  better  explanation  for  the  purposes  of  the  re¬ 
search,  the  principle  of  organizing  the  questionnaire  more  carefully,  and  the  principle 
of  greater  cadet  participation. 
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1.  (jive  cadets  a  nor*  inspirational  Met  of  instructions)  simply  saying,  "The  fol¬ 
lowing  questionnaire  is  designed  to  obtain  research  information  for  the  USMA  Office  of 
Institutional  Research,"  is  not  very  exciting. 

2.  Organise  the  questionnaire  better.  if  the  Office  of  Military  Instruction,  for  ex¬ 
ample,  wants  a  particular  item,  why  not  tell  the  cadets i  "These  items  will  help  OM1  do 
its  job  better"?  By  identifying  who  is  going  to  use  the  information,  it  is  more  likely 
that  they  will  see  the  questionnaires  as  really  useful,  rather  than  a  waste  of  time. 

3.  Schedule  a  feedback  session  for  cadets  who  want  to  learn  more  about  the  results 
of  the  survey.  This  should  increase  cadet  satisfaction  with  the  surveys. 

4.  Involve  selected  cadets  in  planning  the  questionnaires j  this  is  especially  impor¬ 
tant  in  checking  for  completeness  of  coverage,  and  checking  on  the  wording  of  questions. 
Most  professional  survey  researchers  advocate  doing  “pilot  work"  with  a  small  sample. 

Before  mass  sampling  begins,  "pilot  work"  with  small  random  samples  of  cadets  would  ac¬ 
complish  these  objectives. 

5.  When  it  is  not  necessary,  do  not  require  cadots  to  include  their  name  and  cadet 
number  on  the  answer  sheet.38 

6.  Use  the  results  of  the  research,  and  document  the  uses  of  past  research  for  in¬ 
terested  cadets. 

The  fact  that  a  highly  biased,  one-sided  questionnaire  may  cause  a  small  amount  of  atti¬ 
tude  change  should  have  little  direct  impact  on  attitude  research  here  at  USMA.  Kaeh 
research  questionnaire  is  reviewed  intensively  by  researchers  and  by  administrators, 
each  with  their  own  bias,  (liven  the  extensive  review  process,  it  is  unlikely  that  a 
highly  biased,  one-sided  questionnaire  would  be  given  to  cadets. 

Furthermore,  moat  of  the  evidence  on  professionally  constructed  attitude  scales  suggests 
that  they  do  not  promote  attitude  chango.  If  anything,  questionnaires  tend  to  prevent 
attitude  change.  Any  change  in  attitude  or  stabilising  in  attitude  is  likely  to  be  con¬ 
fined  to  unimportant  issues.  Thus,  asking  a  cadet,  "How  do  you  feel  about  Constitution 
Island?"  will  probably  commit  him  to  his  originally  held  opinion.  Asking  him,  "Do  you 
intend  to  resign  from  USMA?"  will  probably  have  virtually  no  effect  on  him,  since  it  is 
a  more  important  question. 

A  final  recommendation  of  this  report  has  to  do  with  the  conduct  of  institutional  re 
search  at  West  Point.  In  any  organisation,  some  research  will  focus  on  immediate  short- 
range  problems  of  the  organisation,  wheroas  other  research  is  focused  on  long-range, 
enduring  problems.  If  a  crisis  occurs,  and  the  organisation  Is  threatened,  naturally 
the  organisation  will  have  to  initiate  some  short-range  institutional  research  to  find 
out  what  caused  the  crisis.  Such  crisis-instigated  research  tends  to  foster  a  faulty 
research  design,  and  an  improper  research  design  leads  to  misinterpretation.  For  ex¬ 
ample,  suppose  thore  is  a  "crisis"  over  resignation:  a  class  is  suspected  of  having  a 
high  number  of  potential  reaignees.  If  the  cadets  are  given  a  questionnaire  with  many 
questions  like,  "Have  you  over  thought  of  resigning?,"  and  many  cadets  do  subsequently 
resign)  then  it  will  seem  as  if  the  questionnaire  caused  the  resignation.  1V>  avoid  such 
post-hoc  fallacies,  the  four  groups  design  (described  in  Table  2;  should  be  used.  To 
avoid  the  faulty  logic  of  crisis-oriented  research,  institutional  research  surveys  should 
be  dono  on  a  planned,  periodic  basis. 

Fortunately,  the  Office  of  Institutional  Research  has  the  capacity  for  dealing  with  po¬ 
tential  long-range  problems  before  they  roach  crisis  proportions.  To  make  the  most  of 
this  capacity,  the  final  recommendation  Is  stated  below i 

7.  Questionnaire  items  should  be  evaluated  not  only  in  terms  of  immediate  practical 
use,  but  also  for  measuring  trends,  and  for  anticipating  long-range  needs,  items  with  no 
“apparent"  immediate  uue  may  bo  extremely  valuable  in  the  long  lun. 


However,  protection  of  Individual  privacy  must  be  weighed  against  accuracy  of  the  in¬ 
format  ton.  Sometimes  anonymity  has  no  effect  in  questionnaire  responses  (Marion,  J., 
"Tiie  Kf feet  of  Anonymity  of  the  Responses  to  the  First  Class  Questionnaire,  Class  of 
1971,”  4E1. 01-72-016,  Sep  ’71) i  sometimes  it  can  lead  to  faking  (Medsger,  (1. ,  “Cadet 
Faking  on  Psychological  Tests."  401. 00-73-025,  May  * 71) . 
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